1. Technical Field
The present invention relates to an image processing technology, and more particularly, to a technology of sharpening an image obtained by, for example, an enlargement process or the like.
2. Background Art
With the widespread use of digital video apparatus networks, it has become common practice to handle digital images in various different standard formats using various input/output apparatuses. Particularly, there are various image sizes ranging from a low resolution to an ultra-high resolution. Examples of low-resolution images include QCIF (176×144 pixels), QVGA (320×240 pixels), CIF (352×288 pixels), and the like, which are used in a camera or a display of a mobile telephone. Examples of standard-resolution images include VGA (640×480), XGA (1024×768), SXGA (1280×1024), and the like, which are sizes for PC displays. Examples of high-resolution images include UXGA (1600×1200), QXGA (2048×1536), HDTV (1920×1080), and the like, which are used in a projector, a special LCD, or the like. Recently, there are ultra-high-resolution images, such as QSXGA (2560×2048), QXGA (2048×1536), QUXGA (3200×2400), and QUXGA-wide (3840×2400), which are used in a display for applications in the medical or printing fields.
It is here assumed that a QVGA (320×240 pixels) image captured by a mobile telephone needs to be displayed with a high definition of QUXGA (3200×2400). In this case, the length and width sizes of an original image are each digitally enlarged by a factor of 10, i.e., an unconventionally high-ratio image enlargement process is required. Conventionally, however, an enlargement ratio assumed in an ordinary digital enlargement process is about 2×2 corresponding to enlargement from the standard TV resolution to the HDTV resolution, and no more than about 4×4 enlargement ratio has been studied (see, for example, Non-patent Document 2).
One-source multi-use of images has penetrated into the broadcast business. Specifically, it is often that only a portion is extracted from a captured image and is used for other applications. For example, when a sport scene (e.g., soccer, etc.) is captured using a wide-angle lens, and an image of an individual player is extracted from the scene and zoomed up for display, an unconventionally high-ratio enlargement process is required. The target value of the enlargement ratio in such an image extracting process has no upper limit.
Thus, image enlargement is a large challenge to digital image processing. The image enlargement technically means that a low-resolution image is transformed into a high-resolution image. The image enlargement, i.e., making higher-resolution, is divided into two categories, depending on whether importance is put on an image input system or an image display system.
The first category is an enlargement process (making higher resolution) in which importance is put on the image input system, corresponding to a so-called “super-resolution” field. Digital super-resolution is achieved by capturing subject information while minutely vibrating an image capturing device or from successive moving images to collect information beyond the sampling limit of the image capturing device, followed by integration and convergence to increase the definition of an original image. This technique is suitable for scientific image measurement in a medical or remote sensing field.
The second category is an enlargement process in which importance is put on the image display system. This technique is intended to transform a blurred image to a visually sharp and preferable image (image sharpening) rather than a high-resolution image faithful to its original image. The technique is employed when an image captured by a camera of a mobile telephone is displayed on a high-resolution display, an standard TV image is displayed on an HDTV screen, or the like. For consumer image apparatuses, an image enlargement process which produces quality tolerable for such high-resolution display is required. The present invention is directed to the process in which importance is put on the image display system.
As conventional techniques in the art, a linear interpolation filter (e.g., bicubic interpolation, etc.), an edge preserving nonlinear filter which enlarges an image while preserving an edge to avoid blurring, and the like, have been studied for many years. However, with these techniques, it is not possible to restore or estimate a high frequency component which a low-resolution image lacks. In order to restore a high frequency component, a technique of using a learning process with low-resolution and high-resolution image samples has been recently studied. Hereinafter, two examples of this technique will be described.
Patent Document 1 discloses a method of generating a high-resolution image from a low-resolution image. Initially, an initial low-resolution image is interpolated or scaled up into a low-resolution image having a desired image size. The resultant low-resolution image is divided into low-resolution patches overlapping each other. For each low-resolution patch, a mid-band patch from which a high frequency component is removed is generated. Thereafter, while scanning the image, a pixel M in a mid-band patch whose contrast is normalized and an adjacent high-band patch H which has already been predicted are linked serially to generate a search vector, and a closest high-band patch is output from a database for training. The high-band patch and the low-resolution patch are combined by addition to successively generate a high-resolution patch having connectivity to an adjacent patch, thereby generating a high-resolution image.
Non-patent Document 1 discloses a technique of sharpening a blurred image by applying wavelet transform. Initially, a sharp image and a degraded image having a blurred edge are subjected to three-stage discrete binary two-dimensional wavelet transform to obtain a 16-dimensional multi-resolution vector for each coordinate in the image. Since only an edge portion in the image is targeted, M learning vectors are used, excluding smoothed components. N representative vectors selected from M learning vectors of a blurred image are listed on an analysis codebook, and N representative vectors generated from M learning vectors of a sharp image are listed on a temporary reproduction codebook. A sharpening process is achieved by looking up the temporary reproduction codebook via a quantization index from the analysis codebook.
Patent Document 1: Japanese Unexamined Patent Publication No. 2003-18398
Non-patent Document 1: Yoshito Abe, Hisakazu Kikuchi, Shigenobu Sasaki, Hiromichi Watanabe, and Yoshiaki Saito, “Edge Enhancement of Images Using Multiresolution Vector Quantization”, IEICE Transactions, Vol. J79A 1996/5 (pp. 1032-1040)
Non-patent Document 2: Makoto Nakashizuka, Hiromichi Ebe, Hisakazu Kikuchi, Ikuo Ishii, and Hideo Makino, “Image Resolution Enhancement on Multiscale Gradient Planes”, IEICE Transactions, D-II, Vol. J81, D-II No. 10 (pp. 2249-2258)
However, there are the following problems with conventional techniques.
Specifically, in the case of techniques using a feature vector obtained by image waveform signal analysis, such as a mid-band frequency vector of a pixel value in an image, a wavelet transform coefficient vector, or the like, image input information is only processed. Therefore, when an enlargement ratio is particularly large, it is difficult to obtain a result much more satisfactory than that of a linear image process, such as a conventional bicubic interpolation technique or the like.
The present inventors consider that it is necessary to accurately incorporate, into image processing, a characteristic of a subject to be captured in an image, such as a material, a distance from a camera, or the like, in addition to image information, in order to more effectively achieve sharpening of an image, i.e., a process of transforming a blurred image into a visually sharp and preferable image.
In addition, by performing such image sharpening after enlargement and interpolation of a low-resolution image, it is possible to generate an enlarged image tolerable for high-resolution display quality from a low-resolution image having a low image size.